Survey and Analysis of Production Distributed Computing Infrastructures

Context The material in this report is a draft of a large part of Chapter 3 of " Abstractions for Distributed Applications and Systems, " a book being written to be published by Wiley in 2012. This report primarily covers production distributed computing infras-tructures that have been used to develop and deploy large-scale scientific applications. We define a production distributed computing infrastructure as a set of computational hardware and software, in multiple locations, intended for use by multiple people who are not the developers of the infrastructure. We observe that typically the time scales over which scientific applications are developed and used is qualitatively larger than the time scales over which the underlying infrastructure tends to evolve. For instance, the middleware used and the services and interfaces offered by many distributed computing infrastructures have changed over recent years due to changes in providers and other technical, political, and funding reasons. Additionally, some of the commercial infrastructures themselves have developed relatively recently. However, one component of this landscape has essentially remained the same: scientific applications and the most commonly used methods used to develop them. The relatively slow evolution of scientific applications is both an opportunity and a challenge. It is a challenge in that once developed , they are hard to modify and adapt to changes in infrastructure. It is an opportunity in the sense that if we can design and architect scientific applications correctly they will be immune to shifts in the underlying infrastructures! Given the many changes in academic computing infrastructures the world over, and the fast evolution of commercial infrastructures, this report is an attempt to provide a topical and focused analysis of distributed computing infrastructures. The book from which this report has originated provides: (i) a critical assessment of a number of existing scientific applications and infrastructures – to identify gaps between application requirements and the abstractions and capabilities provided by the current generation of systems and infrastructure ; (ii) a survey of 13 application case studies; (iii) survey of coordination abstractions and infrastructures currently employed by distributed applications , in particular identifying mechanisms that may have benefit for future applications (in addition to those surveyed); and (iv) a survey and assessment of abstractions and infrastructures within the emerging area of data intensive applications. The book is, in part, a consequence of what we per-1 ceive to be a lack of sufficient connection between: (i) the …